Evaluating and Comparing Possibly Misspecied Forecasts

This paper considers the evaluation of forecasts of a given statistical functional, such as a mean, quantile, or distribution. Recent work has emphasized the importance of evaluating such forecasts using a loss function that is consistent for the functional of interest, of which there are an in…nite number. If forecasters all use correctly speci…ed models, and if the information sets of the competing forecasters are nested, then the construction, evaluation, and comparison of competing forecasts are invariant to the choice of consistent loss function. However, the presence of misspeci…ed models, parameter estimation error, or nonnested information sets, leads to sensitivity to the choice of (consistent) loss function. Thus, rather than merely specifying the target functional, which narrows the set of relevant loss functions only to the class of loss functions consistent for that functional, this paper proposes that forecast consumers or survey designers should specify the single speci…c loss function that will be used to evaluate forecasts. An application to survey forecasts of US in‡ation illustrates the results. Keywords: Survey forecasts, point forecasting, density forecasting, Bregman distance, proper scoring rules, consistent loss functions. J.E.L. codes: C53, C52, E37. AMS 2010 Classi…cations: 62M20, 62P20. I thank Tim Bollerslev, Dean Croushore, Frank Diebold, Tilmann Gneiting, Jia Li and Allan Timmermann for helpful comments. Contact address: Andrew Patton, Department of Economics, Duke University, 213 Social Sciences Building, Box 90097, Durham NC 27708-0097. Email: [email protected].